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DETAILED ACTION 

This action is in response to the amendment filed on 06/05/06. 

Claim Rejections - 35 (JSC § 103 

1 . The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

2. The factual inquiries set forth in Graham v. John Deere Co,, 383 U.S. 1, 148 
USPQ 459 (1966), that are applied for establishing a background for determining 
obviousness under 35 U.S.C. 103(a) are summarized as follows: 

1 . Determining the scope and contents of the prior art. 

2. Ascertaining the differences between the prior art and the claims at issue. 

3. Resolving the level of ordinary skill in the pertinent art. 

4. Considering objective evidence present in the application indicating 
obviousness or nonobviousness. 

3. This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 
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4. Claims 1 -8, 10-17, 19 -25 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over the combination of Grimson et al ("Using adaptive tracking to classify 
and monitor activities in a site", Proceedings 1998 IEEE Conference on Computer 
Vision and Pattern recognition, Santa Barbara, CA, 1998 June 23 - 25; pages 22 - 29) 
and Stein ("Tracking from multiple view points: Self-calibration of space and time", 
Computer Vision and Pattern Recognition, 1999, IEEE Computer Society Conference 
on, Volume 1, 23-25 June 1999 Page(s): 521 - 527). 

In regards to claim 1 1 , Grimson discloses A system for use in monitoring a 
search area, the system comprising: a plurality of imaging devices positioned to cover a 
defined search area, wherein each field of view of each imaging device comprises a 
field of view portion which overlaps with at least one other field of view of another 
imaging device (Section 3); and a computer apparatus operable to (Figures 1 - 8, p. 24 
first three paragraphs under section 3: For a system to perform the operations 
described in this article including creating the images shown through the figures that 
system must include a computer.): fuse image data from the plurality of imaging devices 
into a single image, wherein a plurality of [physically marked landmark points] of 
commonality in field of view portions which overlap is used to fuse image data (Section 
3, Grimson finds homography transform matrices based on moving objects.); segment 
foreground information of the fused image from background information of the fused 
image data (p. 23, second full paragraph in the right column); use the foreground 
information to provide object path data representative of at least one object path of one 
or more moving objects in the search area (p. 23, last three paragraphs: The object path 
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is recorded as location, speed and direction.); recognize one or more defined normal 
and/or abnormal object path feature models based on one or more characteristics 
associated with normal, i.e., common activity, or abnormal, i.e., unusual events, object 
paths of moving objects (p. 26, first paragraph under section 5.1 and first two 
paragraphs under section 5.2); and compare the object path data to the one or more 
defined normal and/or abnormal object path feature models for use in determining 
whether the at least one object path is normal or abnormal (p. 26, first two paragraphs 
under section 5.2). 

Grimson teaches of tracking multiple objects in a "composite image" created 
using homography transform matrices. These matrices are created by tracking moving 
objects in the video streams. Grimson does not teach that a homography matrix should 
be created from the non-moving, i.e., static, objects in the video streams. 

Stein discloses an improvement to the system disclosed by Grimson, by first 
finding (Sections 1.1 and 3.3) a course homography matrix by tracking moving objects 
just as Grimson discloses. Stein notes, "the initial alignment does not perfectly align the 
ground plane". Stein then refines the alignment by finding a homography transform 
matrix in using the static features of the video streams. 

Therefore it would have been obvious to one having ordinary skill in the art at the 
time of the invention to use the refined homography transform approach disclosed by 
Stein in the method taught by Grimson. The combination would allow for a more 
accurate alignment of the video streams. Thus more accurate information would be 
obtained from the composite video stream. 
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In regards to claim 12, The system of claim 1 1 , wherein at least one of the one or 
more characteristics associated with normal or abnormal object paths comprises the 
trajectory thereof (p. 23, last three paragraphs: Trajectory is defined by location, speed 
and direction.). 

In regards to claim 13, The system of claim 11, wherein the one or more defined 
normal and/or abnormal object path feature models comprise one or more defined 
normal object path feature models based on one or more characteristics associated with 
normal object paths, wherein the computer apparatus is further operable to compare the 
object path data to the one or more defined normal object path feature models to 
determine whether the at least one object path is normal, and further wherein the 
system comprises an alarm device operable to provide an alarm if the at least one 
object path is not normal (p. 26, first paragraph under section 5.1 and first two 
paragraphs under section 5.2: "Markpng] for investigation": is a way of alarming a user 
to the fact that an object path is not normal.). 

In regards to claim 14, The system of claim 11, wherein the one or more defined 
normal and/or abnormal object path feature models comprise one or more defined 
threatening, i.e., unusual, and/or non-threatening, i.e., common, object path feature 
models based on one or more characteristics associated with threatening object paths, 
and further wherein the computer apparatus is operable to compare object path data to 
the one or more defined threatening and/or non-threatening object path feature models 
for use in determining whether the at least one object path indicates occurrence of a 
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threatening event (p. 26, first paragraph under section 5.1 and first two paragraphs 
under section 5.2). 

In regards to claim 15, The system of claim 14, wherein the computer apparatus 
is further operable to: provide object path data representative of a plurality of object 
paths corresponding to a plurality of moving objects in the search area over a period of 
time (Figure 2); group the plurality of object paths into one or more clusters based on 
the commonality of one or more characteristics thereof (Figure6, bottom left; p. 27, 
second and third full paragraphs); and identify the one or more clusters as normal object 
path clusters comprising a plurality of object paths representative of normal object paths 
of moving objects in the search area or clusters comprising a single object path or a 
smaller number of object paths relative to the number of object paths in the normal 
object path clusters (p. 28, second full paragraph: By placing clusters into a hierarchy 
common, or normal, object paths are identified.). 

In regards to claim 16, The system of claim 15, wherein the computer apparatus 
is further operable to use the object path data representative of an object path in a 
cluster comprising a single object path or a cluster comprising a smaller number of 
object paths relative to the number of object paths in the normal object path clusters to 
define one or more defined normal and/or abnormal object path feature models (p. 28, 
second full paragraph; p. 26, first paragraph under section 5.1 and first two paragraphs 
under section 5.2). 

In regards to claim 17, The system of claim 15, wherein the computer apparatus 
further is operable to identify the one or more clusters as non-threatening object path 



Application/Control Number: 10/034,761 Page 7 

Art Unit: 2624 

clusters comprising a plurality of object paths representative of non-threatening object 
paths of moving objects in the search area or clusters comprising a single object path or 
a smaller number of object paths relative to the number of object paths in the non- 
threatening object path clusters, and further wherein the computer apparatus is 
operable to determine whether any of the clusters comprising single object paths or the 
smaller number of object paths relative to the number of object paths in the non- 
threatening object path clusters are to be used to define one or more defined 
threatening and/or non-threatening object path feature models for use in determining 
whether an object path indicates occurrence of a threatening event (p. 28, second full 
paragraph; p. 26, first paragraph under section 5.1 and first two paragraphs under 
section 5.2). 

In regards to claim 19, The system of claim 11, wherein the computer apparatus 
is operable to recognize at least one object path tracked in the search area and 
calculate one or more features associated with the at least one object path (p. 23, final 
paragraph). 

In regards to claims 1 - 7 and 10, claims 1 - 7 and 10 are rejected for the same 
reasons as claims 11-17 and 19. The argument analogous to that presented above for 
claims 11-17 and 1 9, is applicable to claims 1 - 7 and 1 0. 

In regards to claim 20, claim 20 is rejected for the same reasons as claim 15. 
The argument analogous to that presented above for claim 15 is applicable to claim 20. 
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In regards to claim 21 - 23, claims 21 - 23 are rejected for the same reasons as 
claim 17. The argument analogous to that presented above for claim 17 is applicable to 
claims 21 -23. 

In regards to claim 25, A system for use in monitoring a moving object in a 
search area, wherein the system comprises: a plurality of imaging devices positioned to 
provide image data covering a defined search area, wherein each field of view of each 
imaging device comprises a field of view portion which overlaps with at least one other 
field of view of another imaging device (p. 24, first three paragraphs under section 3); 
means for fusing all the image data from the plurality of imaging devices into a single 
image using a plurality of [physically marked landmark points] of commonality in field of 
view portions which overlap (p. 24, final paragraph); means for segmenting foreground 
information of the fused image data from background information of the fused image 
data (p. 23, second full paragraph in the right column); means for using the foreground 
information to provide object path data representative of at least one object path of one 
or more moving objects in the search area (p. 23, second full paragraph in the right 
column); means for recognizing one or more defined non-threatening and/or threatening 
object path feature models based on one or more characteristics associated with non- 
threatening and/or threatening object paths of moving objects in the search area (p. 26, 
first paragraph under section 5.1 and first two paragraphs under section 5.2); and 
means for comparing the object path data to the one or more defined non-threatening 
and/or threatening object path feature models for use in determining whether the at 
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least one object path is indicative of a threatening event (p. 26, first paragraph under 
section 5.1 and first two paragraphs under section 5.2). 

Grimson teaches of tracking multiple objects in a "composite image" created 
using homography transform matrices. These matrices are created by tracking moving 
objects in the video streams. Grimson does not teach that a homography matrix should 
be created from the non-moving, i.e., static, objects in the video streams. 

Stein discloses an improvement to the system disclosed by Grimson, by first 
finding (Sections 1.1 and 3.3) a course homography matrix by tracking moving objects 
just as Grimson discloses. Stein notes, "the initial alignment does not perfectly align the 
ground plane". Stein then refines the alignment by finding a homography transform 
matrix in using the static features of the video streams. 

Therefore it would have been obvious to one having ordinary skill in the art at the 
time of the invention to use the refined homography transform approach disclosed by 
Stein in the method taught by Grimson. The combination would allow for a more 
accurate alignment of the video streams. Thus more accurate information would be 
obtained from the composite video stream. 

In regards to claim 24, claim 24 is rejected for the same reasons as claim 25. 
The argument analogous to that presented above for claim 25 is applicable to claim 24. 

Response to Arguments 
1. Applicant's arguments filed 06/05/06 have been fully considered but they are not 
persuasive. 
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2. The applicant first agues "the references cited do not show such selection and 
physical marking". The examiner would like to point out that the applicant is entitled to 
the broadest possible interpretation of the language and thus physical marking can be 
read as any operation used to identify non-moving points of commonality between 
images. Stein discloses that static features are used to form a homography matrix. 
Thus Stein discloses physically marking non-moving points of commonality. 

3. The applicant's primary argument centers on the Irani document supplied by the 
applicant. Although at first blush the applicant's arguments appear to be valid with 
regards to Irani upon closer inspection it is clear that Irani does in fact relay on static 
features and not motion information. Irani is addressing a problem called Ego-motion 
although it would probably be better understood if it were called camera shaking. 
Essentially when a user is holding a video camera, the users hands will shake slightly 
leading to a jittery picture. Irani is trying to use software to correct for this jittering by 
warping the pictures together so the image is smooth. Throughout the article when 
motion is being talked about it is the motion of the camera itself and not the motion of 
the scene. First paragraph of the introduction "The motion observed in an image 
sequence can be caused by camera motion (ego-motion) and by motion of objects 
moving in the scene. In this paper we address the case of the camera moving in a static 
scene. " 

Stein obviously is not dealing with the same situation (a single handheld 
camera). However, after the initial alignment Stein's warped images are rather close to 
what one would expect from ego-motion between frames of video. So Stein's 
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descriptions of the operation are correct. "Given the rough alignment we can use robust 
estimation techniques on the static features to determine a more accurate alignment of 
the ground plane". And, "the general idea is to search for a homography matrix A2 that 
minimizes the 'sum square difference' between image 1 and the warped image 2". So 
to summarize, Irani shows that Stein is using static features to find the difference (using 
homography) between image 1 and warped image 2, this difference is used to refine the 
alignment. 

Conclusion 

4. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Christopher L. Lavin whose telephone number is 571- 
272-7392. The examiner can normally be reached on M - F (8:30 - 5:00). 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Bhavesh M. Mehta can be reached on (571) 272-7453. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 




